Music has been used throughout history in both public and private settings to influence mood and behavior, and to suggest appropriate emotional responses to social stimuli. Such use has given rise to many phenomena such as background music, cinematic soundtracks, and even the scientifically engineered sound offered by Muzak Holdings LLC under the brand Muzak (touted as functional music rather than entertainment because the sound affects those who hear it but does not require a conscious listening effort). Personal introspection will often show that our individually created listening experiences can also be understood by the above functional motivations. Recently, psychological research has demonstrated that listeners choose and respond to music based on internal arousal and emotional states informed by their surroundings. This research implies that music is not an elusive, disembodied experience; rather, music is directly connected to the physicality of everyday experience.
Music has a unique role among the arts. It can act as a suggestive, peripheral backdrop (“aural wallpaper”) or as something in which to actively engage while consistently conveying the functional context of an environment in subtle, non-intrusive ways. Music also stands alone as the only art form that is fully standardized with a universally accepted and ubiquitous digital format, the compact disc or “CD.”
Historically, music consumption has been passive. The listener simply enjoys the sounds of music either directly as played in performance venues or indirectly through technology such as recording studios, radios, portable audio players such as that sold by Sony under the brand “Walkman,” and MP3 players. The original Walkman device became famous for bringing about a change in the listening of music by allowing people to carry their own choice of music with them. “MP3” originally identified an MPEG standard used especially for digitally transmitting music over the Internet, but has come to mean a file containing a song or other audio data that is encoded using this standard. (The Moving Picture Experts Group or “MPEG” is a working group of ISO/IEC charged with the development of video and audio encoding standards.)
More recently, however, interactive systems have broadened the musical experience. One interactive music system is the SensorBox described by J. Allison and T. Place in their article, “SensorBox: Practical Audio Interface for Gestural Performance,” Proceedings of the 2003 Conference on New Interfaces for Musical Expression (NIME-03 (Montreal, Canada). The SensorBox is a system used to input sensor signals into a computer through a digital audio card, with the goal of controlling electronic music performance and composition. Allison and Place describe a method of encoding slowly varying sensor information on audio signals. In order to multiplex the number of sensors they are inputting, carrier waves of different frequencies are used simultaneously, only to be filtered and demodulated by a decoder software. In addition, one of their implementations multiplexes sensor data with real-time musical input. SensorBox is an input device to capture data from gesture-sensors during musical performances to alter sonic parameters within a computer. Thus, it is a tool for a composer rather than a listener. As a performance input device for a computer, SensorBox does not embed the sensor data in an inaudible way.
Another interactive music system was developed by the “Sonic City” project. As disclosed by R. Mazé´ and L. Gaye in “Sonic City,” Proceedings of Cybersonica (2003), the project explores mobile interaction and wearable technology for everyday music creation. The project has designed, implemented, and evaluated a system that creates electronic music based on sensing bodily and environmental factors. Mapping these to the real-time processing of concrete sounds, Sonic City generates a personal soundscape co-produced by physical movement, local activity, and urban ambiance. Encounters, events, architecture, and behaviors—all become ways of interacting with the environment. In this project, the authors' intent is to break out of traditional contexts for music creation to explore creative possibilities within local surroundings and mundane activities. Wearing the Sonic City system, anyone can experience a simple walk down the street as an expressive act and a path through the city as a personal composition. As a complement to lived urban experience, it is an intimate soundscape intended to enhance perception and encourage new uses of the urban landscape.
Thus, Sonic City is a system that is concerned with the connection between the musical experience of listeners (as opposed to a tool for composers) and a multi-sensory environment. Sonic City addresses these details, however, using the environment as input rather than using signals to affect the environment. Like the SensorBox, Sonic City is an input device used to affect composition. The information it uses to generate compositions is the multi-sensory environment of the listener. By sensing light, temperature, the user's movements, and other environmental details, the Sonic City software composes new music on the fly in response to these inputs.
In a press release dated Mar. 9, 2006, Intuitive Devices, Inc. of Los Gatos, Calif. announced an accessory to the popular iPod® device. The iPod® device is a brand of portable media players designed and marketed by Apple Inc. of Cupertino, Calif., and launched in October 2001. As of April 2007, Apple had sold over 100 million units worldwide, making it the best-selling digital audio player series in history. The Blinkit® accessory announced by Intuitive Devices combines a safety blinker with a light beam (akin to a flashlight) and a fun flasher that beats in rhythm to the music played by the iPod® device. The Blinkit® accessory is operated by a single-button, microprocessor-controlled system. The accessory only has a few modes, does not permit selection by the user, has a static functionality, and may be linked solely to the Apple portable media player.
Although the general field of steganography—hiding one signal within another—is large, the subset of the application of steganography to music is much more narrow. Representative of the field of steganography is U.S. Pat. No. 6,947,893, titled “Acoustic Signal Transmission with Insertion Signal for Machine Control” and issued to Iwaki et al. This patent describes an encoding method for embedding a machine control signal in an audio signal that is received and used by an external device. The specific implementation of this technology and its related goals are only peripherally related, however, to music.
Iwaki et al. specifically disclose electrically synthesizing an audible sound signal and another (coding) signal different from the sound signal. A data hiding technique is used under a condition that the auditory presence of the signal cannot be detected by human ears. This signal is emitted from a sound generation device (speaker) to carry out signal output. In perhaps the most relevant application to the present invention, acoustic information (i.e., music) and control signals are transmitted to control sound generation in a robot. Thus, the patent appears to disclose technology used to drive a robot that sings and dances along to a music track.
To overcome the shortcomings of current interactive music systems, the present invention provides a new system. An object of the system is to enrich a listener's musical experience through multi-sensory compositions by providing a method for embedding control data, to drive external devices, within an audio signal in a manner that is compliant with current digital audio standards. A related object is to use digital watermarking or hidden data encoding methods (steganography) to provide a multi-sensory composition in a widely distributed format. Another object is to reintroduce physicality to listening environments using modern technological mechanisms.
It is still another object of the present invention to provide a multi-sensory composition and playback that works within a current audio standard (e.g., the compact disc). An additional object is to combine music listening and the multi-sensory environment in a related and interactive system. Yet another object is to provide an affordable and effective system.